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Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
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2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
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and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 
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contracting  centers  that  support  AMC’s  other  major  subordinate  and  life  cycle  management 
commands.  ACC  provides  global  contracting  support  to  warfighters  through  the  full  spectrum  of 
military  operations.  ACC  consists  of  more  than  5,800  military  and  civilian  personnel  worldwide  who 
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Past  Performance  as  an  Indicator  of  Future  Performance: 
Selecting  an  Industry  Partner  to  Maximize  the  Probability 

of  Program  Success1 

James  Bradshaw — Mr.  Bradshaw  has  more  than  three  decades  of  information  technology 
acquisition  experience  in  the  areas  of  proposal  development,  project  management,  and  purchasing 
management.  He  has  implemented  improved  source  selection  and  project  delivery  processes 
supporting  government  and  commercial  programs.  Mr.  Bradshaw  is  currently  a  senior  economic 
business  analyst  and  group  leader  with  the  MITRE  Corporation,  and  holds  professional  certifications 
in  project  management  (PMP)  and  purchasing  management  (C.P.M.).  [jbradshaw@mitre.org] 

Su  Chang — Su  Chang  has  10  years  of  experience  and  has  worked  for  both  the  DoD  and  civilian 
government  agencies.  She  currently  specializes  in  defense  contracting  and  acquisition  reform 
initiatives.  She  is  a  principal  economic  business  analyst  and  group  leader  with  the  MITRE 
Corporation.  Ms.  Chang  has  been  certified  as  DAWIA  Level  III  in  Contracting  and  is  a  former  1102 
civilian  contract  negotiator,  [sjchang@mitre.org] 

Abstract 

The  federal  contracting  process  should  enable  a  government  organization  to  select  a 
contractor  that  will  become  a  true  business  partner.  Today’s  source  selection  processes 
provide  opportunities  to  evaluate  how  well  a  contractor  proposes  a  solution;  however,  the 
government’s  processes,  policies,  and  tools  are  ill  suited  to  evaluating  how  well  a  contractor 
can  be  expected  to  deliver  on  its  proposed  solutions.  Like  most  government  agencies,  the 
Department  of  Defense  (DoD)  relies  too  heavily  on  the  contractor’s  proposal — what  the 
contractor  claims  it  can  do — versus  evaluating  past  performance  to  determine  what  a 
contractor  has  proven  it  can  do.  The  lack  of  adequate  past  performance  data  and  of 
processes  to  effectively  evaluate  the  qualifications  of  companies,  including  examples  of  the 
contractor’s  trustworthiness  and  key  personnel,  has  contributed  to  a  series  of  program 
failures,  cost  overruns,  and  schedule  delays.  Without  adequate  data  and  processes  to 
address  these  issues,  the  DoD  runs  the  risk  of  duplicating  these  program  failures  and  missing 
the  opportunity  to  capture  this  information  and  prevent  repeated  mistakes  with  the  same 
contractor. 

Executive  Summary 

The  federal  contracting  process  should  enable  a  government  organization  to  select  a 
contractor  that  will  become  a  true  business  partner.  Today’s  source  selection  processes 
provide  opportunities  to  evaluate  how  well  a  contractor  proposes  a  solution;  however,  the 
government’s  processes,  policies,  and  tools  are  ill  suited  to  evaluating  how  well  a  contractor 
can  be  expected  to  deliver  on  its  proposed  solutions.  Like  most  government  agencies,  the 
Department  of  Defense  (DoD)  relies  too  heavily  on  the  contractor’s  proposal — what  the 
contractor  claims  it  can  do — versus  evaluating  past  performance  to  determine  what  a 
contractor  has  proven  it  can  do.  As  selecting  the  right  contractor  is  perhaps  the  most 
important  contributor  to  program  success,  this  report  discusses  the  following  three  problem 
areas  and  recommends  approaches  to  improve  the  probability  of  a  successful  program 
outcome. 

Emphasis  on  evaluating  the  contractor’s  proposed  solution  versus  evaluating 
the  contractor’s  record  of  performance.  During  the  source  selection  process  the 
government  evaluates  a  contractor’s  proposal,  which  consists  of  cost,  technical,  and  past 

1  The  views,  opinions,  and/or  findings  contained  in  this  report  are  those  of  The  MITRE  Corporation  and  should 
not  be  construed  as  an  official  government  position,  policy,  or  decision,  unless  designated  by  other 
documentation.  Approved  for  Public  Release  12-1663.  Distribution  unlimited.  ©2012  The  MITRE  Corporation.  All 
rights  reserved. 
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performance  volumes.  In  most  cases,  however,  past  performance  has  little  or  no  impact  on 
the  final  selection.  Instead,  this  aspect  often  is  minimized  as  a  superficial  pass/fail  test  rather 
than  being  viewed  as  an  effective  tool  to  predict  how  well  a  contractor  will  perform  in  the 
future.  Without  objective  information  on  past  performance  and  processes  to  evaluate 
contractor  qualifications,  government  agencies  are  reluctant  to  discriminate  among 
proposals  based  on  past  performance,  especially  given  the  risk  of  protest.  Alpha  contracting 
negotiations  with  pre-qualified  vendor  candidates  and  product  demonstrations  as  part  of  the 
source  selection  offer  potential  solutions  to  mitigate  this  evaluation  risk. 

Lack  of  consistent  and  thorough  processes  to  evaluate  proposed  key 
contractor  personnel.  The  personnel  on  the  contractor  program  team  make  vital 
contributions  to  successful  program  execution,  and,  as  such,  their  qualifications  should  be 
an  important  evaluation  factor.  Contractors  generally  identify  key  personnel  and  include  their 
resumes  as  part  of  the  proposal  package,  but  most  source  selections  limit  evaluation  of 
these  key  personnel  to  a  simple  determination  of  whether  the  proposed  team  members 
meet  experience  criteria.  The  DoD  could  augment  this  evaluation  by  including  oral 
presentations  in  the  evaluation  process  and  using  the  Q&A  process  to  verify  the 
qualifications  and  experience  cited  in  proposed  key  personnel  resumes. 

Inadequacy  of  government  repositories  of  past  performance  data.  The  ability  to 
evaluate  past  performance  effectively  depends  heavily  on  the  availability  of  tools  and  the 
quality  of  data.  The  government  is  currently  improving  and  consolidating  its  repositories  of 
past  performance  data;  however,  standards  for  capabilities  and  data-entry  requirements  are 
inconsistently  enforced  across  the  DoD.  To  address  these  shortcomings,  the  DoD  should 
issue  a  policy  mandate  that  enforces  data-entry  standards  and  use  an  online  tool  to  track 
and  report  compliance  with  this  policy  mandate.  Data-repository  tools  should  also  be 
improved  to  include  quantifiable  metrics  as  new  data  fields  and  provide  capabilities  to 
upload  validated  contract  deliverables. 

Introduction 

Selecting  the  right  contractor  is  perhaps  the  most  critical  factor  in  achieving  future 
program  success.  This  research  identifies  processes  and  proposes  strategies  that  could 
strengthen  how  the  government,  specifically  the  Department  of  Defense  (DoD),  selects  an 
industry  partner  to  maximize  the  probability  of  program  success. 

Ideally,  the  relationship  between  the  government  and  a  contractor  should  be  a 
genuine  business  partnership.  When  awarding  a  contract  for  a  major  weapon  system  or 
program,  especially  one  that  requires  significant  development,  the  DoD  must  expect  the 
relationship  to  be  a  long-term  commitment.  The  contractor  and  government  program 
management  office  (PMO)  each  play  numerous  critical  roles  throughout  the  program 
planning,  executing,  and  reporting  cycles.  In  a  successful  program,  a  contractor  organization 
and  the  PMO  work  in  harmony  to  strive  toward  program  success.  They  understand  and 
leverage  each  other’s  strengths:  they  communicate  frequently;  they  collectively  manage  the 
vast  web  of  program  stakeholders;  and  they  collectively  establish  the  reputation  and 
credibility  of  the  program. 

The  government  must,  therefore,  strive  to  enter  into  business  relationships  that 
provide  high  confidence  of  success.  Selecting  an  experienced  and  qualified  business 
partner  is  a  critical  step  in  achieving  this  objective.  While  evaluation  of  the  contractor’s 
proposal  is  important,  this  review  can  provide  only  a  prediction  of  what  may  happen  and  a 
description  of  how  the  contractor  plans  to  perform.  By  contrast,  the  contractor’s  experience 
and  past  performance  offer  objective  measures  of  what  the  firm  has  accomplished  in  the 
past. 
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The  paper  identifies  three  key  problem  areas: 

1 .  The  DoD  evaluates  the  contractor’s  proposed  solution  rather  than  the 
contractor’s  record  of  performance.  During  the  source  selection  process,  the 
DoD  evaluates  a  contractor’s  proposal,  which  consists  of  cost,  technical,  and 
past  performance  volumes.  In  most  cases,  however,  past  performance  has 
little  or  no  impact  on  the  final  selection  of  a  contractor.  Instead,  this  aspect 
often  is  minimized  as  a  superficial  pass/fail  test  rather  than  being  viewed  as 
an  effective  tool  to  predict  how  well  a  contractor  will  perform  in  the  future. 
Without  objective  information  on  past  performance  and  processes  to  evaluate 
contractor  qualifications,  the  DoD  is  reluctant  to  discriminate  among 
proposals  based  on  past  performance,  especially  given  the  risk  of  protest. 

2.  The  DoD  lacks  consistent  and  thorough  processes  to  evaluate  key  personnel. 
The  personnel  on  the  contractor  program  team  make  vital  contributions  to 
program  success,  and,  as  such,  their  qualifications  should  be  an  important 
evaluation  factor.  Contractors  generally  identify  key  personnel  and  include 
their  resumes  as  part  of  the  proposal  package,  but  most  source  selections 
limit  evaluation  of  these  key  personnel  to  a  simple  determination  of  whether 
the  proposed  team  members  meet  the  experience  criteria. 

3.  The  DoD  lacks  adequate  tools  to  collect,  analyze,  and  report  past 
performance  information  consistently  across  contracts.  The  ability  to  evaluate 
past  performance  effectively  depends  heavily  on  the  availability  of  tools  and 
the  quality  of  data  content.  The  government  is  currently  improving  and 
consolidating  its  repositories  of  past  performance  data;  however,  the  DoD 
lacks  capabilities  for  information  collection  and  has  not  consistently  enforced 
data  entry  requirements. 

These  problem  areas  represent  three  different  deficiencies  in  the  evaluation  of  a 
contractor’s  past  performance.  Evaluation  is  hampered  by  the  lack  of  adequate  tools  to 
collect,  analyze,  and  report  relevant  information.  The  lack  of  processes  to  assess  the 
capabilities  and  experience  of  key  personnel  puts  at  risk  the  government’s  ability  to  predict 
future  performance  on  a  contract.  In  addition,  the  ability  to  collect,  analyze,  and  report 
information  on  past  performance  is  impaired  by  the  lack  of  policies  and  processes  to 
incentivize  better  reporting.  Together,  these  shortcomings  affect  the  overall  contracting 
process  and  contribute  to  a  cycle  of  inadequacies  that  has  resulted  in  the  numerous 
contracting  challenges  we  face  today.  The  DoD  needs  to  take  a  comprehensive,  coherent 
approach  to  resolve  these  problems  to  achieve  overall  past  performance  improvements. 

Background 

With  the  passage  of  the  Federal  Acquisition  Streamlining  Act  of  1994,  Congress 
acknowledged  that  it  is  appropriate  and  relevant  for  the  government  to  consider  a 
contractor’s  past  performance  in  evaluating  whether  that  contractor  should  receive  future 
work  (Inspector  General,  2008,  p.  i). In  response,  federal  departments  initiated  procedures 
and  systems  to  record  and  use  information  on  past  contractor  performance  during  source 
selection.  Although  the  government  recognizes  that  systematically  documenting  the 
contractor’s  performance  becomes  a  powerful  motivator  for  a  contractor  to  sustain  high- 
quality  efforts,  a  consistent  process  for  recording  performance  has  been  difficult  to 
instantiate  and  maintain  (Office  of  Federal  Procurement  Policy  [OFPP],  Office  of 
Management  and  Budget  [OMB],  Executive  Office  of  the  President,  2000,  p.  1). 

In  May  2010,  the  OFPP  directed  government  agencies  to  integrate  past  performance 
data  systems  in  order  to  provide  consistency  and  ensure  reliability  of  data  across  the  federal 
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government.  The  Acquisition  Committee  for  eGovernment  (ACE)  determined  that  the  entire 
federal  government  will  use  the  DoD  Contractor  Performance  Assessment  Reporting 
System  (CPARS)  as  the  single  system  for  collecting  and  transmitting  performance 
evaluations  to  the  Past  Performance  Information  Retrieval  System  (PPIRS,  n.d.).  The  Navy 
created  CPARS  in  1998  to  meet  information  requirements  established  by  the  Federal 
Acquisition  Regulation  (FAR).  Due  to  inconsistent  CPARS  data  collection  and  reporting,  the 
DoD  published  a  best  practices  guide  titled  A  Guide  to  Collection  and  Use  of  Past 
Performance  Information  in  May  2003.  In  June  2007,  the  DoD  issued  another  CPARS  Policy 
Guide  that  required  all  new  contracts  within  a  certain  dollar  threshold  to  register  in  CPARS 
within  30  days  of  contract  award  (Inspector  General,  2008,  p.  5). 

On  January  21 , 201 1 ,  the  former  OFPP  Administrator,  Daniel  Gordon,  sent  a 
memorandum  to  chief  acquisition  officers  and  senior  procurement  executives  on  enhancing 
assessments  of  contractor  past  performance,  which  included  steps  and  strategies  for 
improving  the  collection  of  information  on  past  performance.  The  memorandum  (Gordon, 

201 1 )  emphasized  that  while  compliance  with  reporting  requirements  is  important,  the 
quality  of  the  reports  submitted  is  far  more  crucial.  It  also  acknowledged  various  challenges 
that  contribute  to  the  low  number  and  quality  of  these  assessments,  which  include  staff 
shortages  and  the  transition  to  the  new  federal-wide  system  that  integrates  the  PPIRS, 
CPARS,  and  the  Federal  Awardee  Performance  and  Integrity  Information  System  (FAPPIS). 

Despite  efforts  by  the  DoD  and  other  government  organizations,  all  of  the  policies 
and  mandates  lack  enforcement  mechanisms  to  incentivize  better  behavior  and  improve 
past-performance  practices.  This  paper  recommends  solutions  that  will  strengthen  the  data, 
processes,  and  tools  associated  with  evaluating  past  performance  and  provide  the 
government  with  a  strategy  to  benefit  from  overall  improvements  in  past  performance 
processes,  policies,  and  tools. 

Problem:  The  DoD  Evaluates  the  Contractor’s  Proposed  Solution  Versus 
Evaluating  the  Contractor’s  Record  of  Performance 

FAR  (2010)  Part  15.3  covers  the  selection  of  a  contractor  in  competitive  negotiations. 
According  to  the  FAR  (2010),  the  government  is  required  to  evaluate  three  areas  for 
acquisitions:  the  quality  of  the  proposed  product  or  service,  the  price  or  cost  to  the 
government,  and  past  performance  (for  acquisitions  that  exceed  the  simplified  acquisition 
threshold).  The  government  often  requests  the  contractor  to  submit  volumes  covering 
technical,  cost,  and  past  performance  as  part  of  its  proposal.  The  technical  proposal 
conveys  the  contractor’s  proposed  technical  solution  or  response  to  a  requirement;  the  cost 
proposal  identifies  the  proposed  costs  of  delivering  the  proposed  technical  solution;  and  the 
past  performance  volume  shows  how  the  contractor  performed  on  previous  similar  or  related 
efforts  as  an  early  indicator  of  potential  future  performance. 

Unfortunately,  the  DoD  currently  lacks  both  adequate  data  on  past  performance  and 
effective  processes  to  evaluate  a  company’s  qualifications,  including  key  personnel.  Thus, 
evaluation  of  past  performance  is  often  not  used  effectively  to  predict  future  performance  on 
a  contract.  These  conclusions  are  borne  out  by  a  2009  study  by  the  U.S.  Government 
Accountability  Office  (GAO).  The  study  examined  past  performance  practices  by  analyzing 
62  procurements  in  five  agencies  that  perform  major  acquisitions:  the  DoD,  the  Department 
of  Energy  (DOE),  the  GSA,  the  Department  of  Homeland  Security  (DHS),  and  the  National 
Aeronautics  and  Space  Administration  (NASA).  The  study  (GAO,  2009,  pp.  8-9)  found  that 
82%  of  the  past  performance  evaluations  did  not  contain  narratives  sufficiently  detailed  to 
establish  that  the  resulting  ratings  were  credible  or  justifiable. 
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In  many  of  these  acquisitions,  the  government  cited  a  company’s  technical  approach 
as  the  most  important  non-cost  factor.  More  than  60%  of  the  contracting  officers  “stated  that 
past  performance  is  rarely  or  never  a  deciding  factor  in  selecting  a  contractor”  (GAO,  2009, 
pp.  8-9).  However,  the  acquisitions  that  placed  emphasis  on  past  performance  noted  that 
this  encouraged  companies  to  perform  better  (GAO,  2009,  pp.  2-3).  The  report  succinctly 
concluded,  “Regardless  of  the  source  used,  contracting  officials  agreed  that  for  past 
performance  information  to  be  meaningful  in  contract  award  decisions,  it  must  be 
documented,  relevant,  and  reliable”  (GAO,  2009,  pp.  8-9). 

Without  such  information,  the  government  must  rely  heavily  on  other  evaluation 
factors  (primarily  technical  and  cost)  that  describe  a  hypothetical  situation:  how  the 
contractor  plans  to  perform  on  a  contract.  In  essence,  this  means  that  evaluations  are  based 
on  the  quality  of  a  document  produced  by  a  professional  proposal  writing  team  during  the 
45-90  day  solicitation  timeframe.  These  teams  specialize  in  showcasing  their  company’s 
capabilities;  they  are  not  necessarily  experts  in  devising  solutions  that  lie  within  the 
company’s  technical  capabilities  and  that  the  company  can  execute  with  acceptable  risk. 
Proposals  are  often  written  with  little  or  no  input  from  the  staff  members  who  will  execute  the 
day-to-day  contract  requirements.  As  a  result,  proposals  submitted  for  competitive  source 
selections  often  present  “optimistic”  solutions  that  carry  significant  inherent  risk.  This  risk 
often  materializes  during  contract  execution,  which  directly  contributes  to  the  program 
failures,  cost  overruns,  and  schedule  delays  that  are  endemic  to  DoD  acquisitions.  The 
heavy  scrutiny  of  technical  and  cost  proposal  evaluations  can  prove  a  wasted  exercise  when 
contractors  continually  overrun  budgets  or  require  major  engineering  change  proposals 
(ECPs)  throughout  the  life  of  the  acquisition. 

Given  the  shortcomings  described  previously,  the  evaluation  of  past  performance 
often  becomes  a  superficial  pass/fail  test.  The  majority  of  best  value  source  selections 
require  the  government  to  evaluate  past  performance  on  the  basis  of  a  sliding  scale  that 
ranges  from  high  confidence  to  no  confidence.  The  DoD  uses  this  scale  to  assess  its  level  of 
confidence  in  a  contractor’s  ability  to  perform  based  on  previous  related  experience.  Without 
adequate  information  about  all  offerors,  the  DoD  is  reluctant  to  place  strong  emphasis  on 
past  performance,  especially  due  to  the  high  risk  of  protest.  As  a  result,  the  DoD  usually 
assigns  all  offerors  the  same  confidence  rating  in  order  to  “level  the  playing  field”  and  uses 
other  evaluation  factors  to  discriminate  among  proposals.  Not  surprisingly,  the  government 
often  fails  to  select  a  business  partner  with  an  established  history  of  high  performance  and 
relevant  experience. 

The  OFPP  (1997)  conducted  a  pilot  test  with  30  contracts  that  used  past 
performance  as  a  significant  evaluation  factor.  Those  contracts  identified  a  20%  increase  in 
average  customer  satisfaction,  confirming  the  assertion  that  the  increased  use  of  past 
performance  provides  improved  results  (OFPP,  1997,  p.  2).  To  improve  how  the  DoD 
leverages  past  performance  in  its  source  selections,  the  remainder  of  this  section  reviews 
proposed  solutions  that  leverage  well-founded  best  practices  to  mitigate  the  risks  associated 
with  inadequate  past  performance  evaluations. 

Proposed  Solutions 

Alpha  Contracting  Negotiations  With  Pre-Qualified  Vendor  Candidates 

Alpha  contracting  negotiations  with  pre-qualified  vendors  leverage  the  best  practices 
of  the  Qualifications-Based  Selection  (QBS)  strategy  authorized  for  Architecture  and 
Engineering  (A/E)  contracts  under  FAR  (2010)  Part  36.  The  QBS  strategy  was  originally 
developed  because  certain  creative  professional  services  cannot  be  fairly  priced  before  the 
creative  process  has  taken  place.  The  QBS  process  improves  innovation,  quality,  and 


ACQUISITION  RESEARCH  PROGRAM: 

CREATING  SYNERGY  FOR  INFORMED  CHANGE  -  342  - 


creativity  by  allowing  the  government  to  negotiate  the  contract  requirements  and  pricing 
jointly  with  the  contractor.  The  Brooks  Act  of  1972  delineates  the  framework  for  QBS  using 
the  following  phased  approach: 

1 .  The  government  determines  the  desired  qualifications,  including  the 
experience  and  demonstrated  competence  of  interested  parties.  The 
information  provided  by  the  offerors  should  emphasize  technical  ability  (cost 
is  not  considered  as  part  of  the  initial  evaluation). 

2.  The  government  creates  a  short  list,  ranking  the  bidders  by  their 
qualifications. 

3.  The  government  conducts  interviews  with  the  firms  on  the  short  list  and  then 
re-ranks  the  firms. 

4.  Finally,  the  government  negotiates  a  statement  of  work  (SOW)  and  fair  price 
with  the  most  qualified  firm. 

Unfortunately,  the  FAR  only  allows  use  of  QBS  for  contracts  involving  the 
construction  or  building  trades,  including  transportation  systems.  However,  the  DoD  could 
leverage  some  of  the  benefits  offered  by  the  A/E  procurement  strategy  by  combining  some 
QBS  attributes  with  other  contracting  best  practices  and  strategies.  For  example,  the  DoD 
can  use  a  prequalification  process  to  narrow  the  pool  of  qualified  vendors,  similar  to  step  1 
in  the  QBS  process.  This  can  be  done  by  using  an  advisory  multi-step  process  described  in 
FAR  (2010)  Part  15.202,  or  by  establishing  a  competitive  range  based  entirely  on  past 
performance  criteria.  Narrowing  the  field  of  vendor  candidates  would  ensure  that  the  DoD 
optimizes  its  time  and  resources  by  conducting  Request  for  Proposal  (RFP)  activities  only 
with  the  vendors  that  have  the  highest  likelihood  of  being  selected  for  contract  award.  This 
approach  also  benefits  potential  contractors,  because  they  do  not  have  to  expend  valuable 
proposal  development  funds  for  solicitations  they  have  little  chance  of  winning. 

Processes  to  narrow  down  the  pool  of  qualified  vendors  on  the  basis  of  past 
performance  and  experience  can  require  upfront  time  and  investment  from  the  government. 
However,  these  strategies  can  save  significant  time  during  the  “back  end”  of  the  proposal 
evaluation.  Ranking  the  vendors  also  helps  keep  competitive  pressures  in  play  during 
negotiations.  The  government  mitigates  the  disadvantages  associated  with  a  sole-source 
negotiation  environment  because  it  maintains  the  flexibility  to  move  to  the  next-ranked 
vendor  at  any  point  in  the  negotiations. 

Additionally,  the  DoD  can  use  a  Q&A  session  as  part  of  an  oral  presentation 
strategy,  similar  to  the  QBS  interview  process  described  in  Step  3  above  and  in  the  Evaluate 
Key  Personnel  Using  Oral  Presentations  section  of  this  paper.  Open  communication  with 
vendors  to  verify  experience  and  qualifications  is  an  important,  but  often  overlooked,  activity. 
Usually  the  government  accepts  the  qualifications  and  experience  stated  in  a  company’s 
proposal  at  face  value.  However,  greater  insight  regarding  the  contractor’s  related 
experience  and  the  qualifications  of  the  proposed  key  personnel  would  increase  the 
accuracy  of  the  government’s  assessment. 

Lastly,  the  DoD  can  apply  the  same  SOW  and  contract  price-negotiation  process 
described  in  Step  4  of  the  QBS  through  the  use  of  alpha  contracting,  an  innovative  strategy 
that  allows  the  government  to  perform  many  activities  jointly  with  the  contractor.  Together, 
the  government  and  contractor  develop  the  SOW  and  proposal  in  a  streamlined  fashion. 

This  process  offers  a  number  of  advantages  and  performance  improvements,  such  as 
enhancing  communication,  refining  and  clarifying  requirements,  and  ensuring  the  technical 
solution  is  bounded  by  the  capabilities  of  the  contractor.  The  alpha  contracting  process  also 
has  the  potential  to  yield  significant  savings  in  time  and  cost.  For  example,  the  Army  PM- 
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Tank  Main  Armament  Systems  M830A1  procurement  utilized  the  alpha  contracting  process 
and  reduced  the  lead-time  for  procurement  administration  by  55%,  thus  saving  $1  million, 
which  was  subsequently  used  to  buy  additional  units  (Jones,  n.d.). 

It  may  be  possible  to  combine  all  of  the  above  techniques  and  use  a  General 
Services  Administration  (GSA)  Blanket  Purchase  Agreement  (BPA)  to  model  the  QBS 
strategy  from  Steps  1-4.  Under  this  arrangement,  the  DoD  could  select  a  set  of  contractors 
from  a  GSA  schedule  and  use  a  prequalification  process  to  narrow  the  field  of  vendors. 

Next,  the  DoD  would  use  a  Q&A  process  to  interview  candidates  and  rank  order  the 
potential  vendors.  The  DoD  could  then  utilize  the  alpha  contracting  process  to  negotiate  a 
BPA  with  the  highest  ranked  firm.  If  negotiations  with  the  top-ranked  vendor  resulted  in 
unfair  pricing,  or  if  the  DoD  and  the  vendor  were  unable  to  agree  on  an  SOW,  the  DoD 
would  have  the  right  to  conduct  negotiations  with  the  second-ranked  firm.  When 
negotiations  were  complete,  the  DoD  would  issue  a  BPA  using  a  GSA  schedule  contract 
authorized  under  FAR  (2010)  Part  8.405-3. 

This  GSA  BPA  approach  has  certain  restrictions  that  may  limit  applicability  across  a 
broad  range  of  acquisitions  (e.g.,  GSA  BPAs  are  limited  to  commercial  products  and 
services  and  do  not  allow  cost-type  contracts  [FAR,  2010,  Subpart  13.303]).  Furthermore, 
the  QBS  approach  is  currently  limited  to  A/E  contracts  authorized  under  FAR  (2010)  Part  36. 
However,  the  need  for  creativity  and  innovation  is  not  limited  to  A/E  procurements.  Agile 
software  development,  for  example,  values  flexibility,  innovation,  and  collaboration  and  can 
greatly  benefit  from  a  QBS-like  acquisition  approach. 

As  a  result,  the  DoD  should  pursue  an  amendment  to  FAR  (2010)  Part  15  to  allow  a 
QBS-like  process  that  can  be  leveraged  for  many  different  types  of  acquisitions.  The  DoD 
could  achieve  maximum  benefit  from  this  approach  if  the  amendment  allowed  a 
prequalification  process  to  narrow  the  field  of  potential  vendors.  An  interview  or  Q&A 
process  could  facilitate  the  vendor  ranking  process,  and  alpha  contracting  would  allow  the 
DoD  to  negotiate  an  SOW  and  pricing  based  on  the  ranking  of  firms.  All  of  these  individual 
strategies  are  permitted  under  different  areas  of  the  FAR;  however,  a  FAR  Part  15 
amendment  would  allow  the  DoD  to  combine  them  into  a  single  process  to  obtain  maximum 
benefit. 


Product  Demonstrations  as  a  Source  Selection  Technique 

The  limited  availability  of  past  performance  information  makes  it  difficult  to  verify  the 
relevance  of  previous  experience.  As  an  alternative  strategy,  the  DoD  could  use 
demonstrations  as  part  of  the  source  selection  process  to  validate  previous  experience.  A 
demonstration  could  include  any  material  representation  of  past  experience  (e.g., 
prototypes,  software  demonstrations).  Using  this  strategy,  the  DoD  would  require  vendors  to 
submit  a  functioning  prototype,  or  a  previously  developed  relevant  product,  and  to 
demonstrate  its  capabilities  as  part  of  the  proposal  process.  The  DoD  could  apply  this 
strategy  to  both  hardware  and  software  procurements;  for  example,  screen  mock-ups  and 
software  demonstrations  from  previous  efforts  could  be  used  to  evaluate  certain  software 
procurements.  The  demonstration  could  be  submitted  prior  to  the  formal  proposal  process, 
or  could  be  used  to  supplement  a  written  technical  proposal.  The  proposal  instructions 
would  specify  the  desired  functionality  of  the  demonstrated  product,  and  thoroughly  describe 
how  the  DoD  would  evaluate  the  demonstration. 

This  source  selection  strategy  will  not  only  help  to  narrow  the  pool  of  qualified 
vendors,  but  will  also  offer  a  way  for  vendors  to  establish  recent  and  relevant  experience. 
The  demonstration  becomes  a  tangible  piece  of  evidence  that  the  contractor  has  verified 
and  proven  its  experience  with  the  technology  and/or  solution  proposed. 
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Problem:  The  DoD  Lacks  Consistent  and  Thorough  Processes  to  Evaluate  Key 
Personnel 

The  overall  experience  and  business  processes  of  the  contractor  are  central  to 
evaluation  of  past  performance.  However,  the  qualifications  of  the  personnel  proposed  for 
key  roles  on  the  contract  have  even  greater  importance  for  the  future  success  of  the 
program.  The  capabilities  of  the  key  personnel  proposed  should  reflect  the  strength  of  the 
company’s  experience. 

DoD  processes  should  aspire  not  only  to  select  the  strongest  contractor,  but  also  to 
ensure  that  the  contractor  assigns  the  “A  Team”  to  the  program.  When  companies  know 
their  key  personnel  will  be  closely  evaluated,  they  have  an  incentive  to  offer  their  best 
performers,  rather  than  simply  personnel  who  meet  minimal  qualification  requirements.  Top¬ 
performing  personnel  with  a  proven  track  record  will  also  be  inclined  to  ensure  the  technical 
solution  proposed  can  be  executed  with  acceptable  risk,  potentially  providing  internal  checks 
and  balances  with  the  proposal  writing  team. 

Today,  contractors  generally  identify  key  personnel  and  include  their  resumes  as  part 
of  the  proposal  package,  but  most  source  selections  limit  evaluation  of  these  key  personnel 
to  a  simple  check  of  whether  the  proposed  team  members  meet  experience  criteria.  The 
DoD  should  have  consistent  and  thorough  processes  for  the  evaluation  of  key  personnel 
that  go  beyond  a  checklist  of  experience  criteria.  This  paper  recommends  that  the  DoD 
increase  the  use  of  oral  presentations  to  verify  the  qualifications  of  key  personnel. 

Proposed  Solution 

Evaluate  Key  Personnel  Using  Oral  Presentations 

Throughout  the  government,  oral  presentations  are  often  used  in  conjunction  with 
written  proposals  to  clarify  or  support  all  or  some  aspect  of  the  contractor’s  technical  and/or 
management  proposal  (Sade,  n.d.).  These  presentations  can  greatly  enhance  the  evaluation 
process.  They  give  the  government  an  opportunity  to  evaluate  the  effectiveness  of  the 
contractor’s  communication  style,  clarify  the  contractor’s  proposal,  and  assess  the  group 
dynamics  of  the  contractor  team.  In  addition,  oral  presentations  remove  the  gloss  created  by 
professional  proposal  writers  and  give  the  government  an  opportunity  to  interact  directly  with 
the  contractor  team  that  will  perform  the  day-to-day  tasks  required  in  the  contract. 

The  DoD  should  take  advantage  of  this  opportunity  to  interact  directly  with  key 
vendor  personnel.  To  further  enhance  the  benefits  of  these  interactions,  the  DoD  should 
mandate  that  proposed  key  personnel  present  the  oral  proposals.  This  prevents  contractors 
from  using  their  best  marketing  representatives  to  make  presentations  and  gives  the 
government  an  opportunity  to  evaluate  the  overall  strength  of  the  contractor  team  that  will 
perform  the  actual  program  tasks. 

In  addition,  the  DoD  should  use  the  Q&A  portion  of  presentation  to  verify  the 
qualifications  and  experience  of  key  personnel.  The  DoD  can  review  the  resumes  of  the 
proposed  personnel  ahead  of  time,  and  prepare  targeted  questions  that  will  help  the 
government  understand  the  breadth  and  depth  of  proposed  key  personnel.  The  government 
should  structure  questions  according  to  behavioral  interviewing  techniques:  asking 
questions  that  require  the  candidates  to  describe  a  past  situation  in  which  they  exhibited  a 
certain  attribute.  This  strategy  is  used  by  more  than  70%  of  Fortune  500  companies,  which 
credit  it  with  being  55%  predictive  of  future  on-the-job-behavior  (Hansen,  n.d.). 

The  DoD’s  evaluation  of  key  personnel  should  go  beyond  verifying  a  checklist  of 
experience  criteria.  Key  considerations  must  include  formal  assessments  of  the  contractor’s 
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trustworthiness  with  respect  to  foreign  influence  and  control,  the  rigor  of  corporate  security 
protocols,  and  the  integrity  of  the  contractor’s  global  supply  chain  relationships. 

Lastly,  the  contract  should  include  key  personnel  clauses  that  protect  the 
government  from  “bait-and-switch”  tactics.  In  some  instances,  contractors  have  replaced  the 
team  originally  proposed  with  potentially  less  qualified  candidates  shortly  after  contract 
award.  To  mitigate  this  risk,  DoD  contracts  should  require  that  the  key  personnel  originally 
proposed  on  the  contract  remain  with  the  program  for  a  minimum  time  period  (e.g.,  12-18 
months),  unless  that  individual  leaves  the  company.  This  becomes  especially  important  if 
the  government  uses  key  personnel  as  an  important  evaluation  factor  in  the  source  selection 
process. 

Problem:  The  DoD  Lacks  the  Adequate  Tools  and  Information  to  Collect, 
Analyze,  and  Report  Past  Performance  Information  Consistently  Across 
Contracts 

A  high-quality  data  repository  greatly  strengthens  the  government’s  ability  to 
evaluate  past  performance  as  part  of  the  source  selection  process.  Well-documented 
information,  easily  accessible  in  such  repositories,  enables  the  government  to  defend  its 
position  during  a  contract  protest.  Thus,  a  data-driven  process  to  collect,  analyze,  and  report 
past  performance  information  is  critical  in  allowing  the  government  to  incorporate  past 
performance  in  a  source  selection  decision. 

As  noted  in  the  Introduction,  two  important  past  performance  tools  used  today  are 
CPARS  and  PPIRS.  CPARS  is  a  suite  of  web-enabled  applications  that  document 
contractors’  past  performance  in  accordance  with  the  FAR.  It  uses  an  electronic  workflow  to 
automate  contracting  officers’  evaluations  of  contractor  performance.  It  also  allows 
contractors  to  submit  comments  electronically  in  response  to  a  government  assessment  and 
either  concur  or  non-concur.  The  DoD-developed  PPIRS  is  a  web-enabled  government 
enterprise  application  supporting  source  selections.  Since  the  FAR  requires  all  federal 
agencies  to  post  contractor  performance  evaluations  in  PPIRS,  it  now  acts  as  the 
government-wide  warehouse  for  information  on  contractor  past  performance  (PPIRS,  n.d.).2 
As  a  warehouse,  it  allows  query-based,  read-only  retrieval  and  review  of  contractor  data  by 
qualified  government  acquisition  personnel.  Contractors  can  view  their  own  data  by  using  a 
central  contractor  registration  to  gain  access. 

Although  the  FAR  requires  the  government  to  evaluate  past  performance,  the 
collection  and  reporting  of  past  performance  data  has  been  inconsistent  and  often  untimely. 
The  lack  of  consistent,  available,  and  reliable  data  hinders  the  DoD’s  ability  to  make  past 
performance  an  effective  part  of  the  proposal  evaluation  process.  The  government  has 
recently  made  improvements  to  many  of  the  past  performance  tools,  including  CPARS  and 
PPIRS,  to  provide  commonality  of  data  entry  format  and  increase  the  availability  of  data  for 
retrieval.  Despite  these  improvements,  current  data  repositories  still  perform  their  functions 
inadequately,  or  are  used  inaccurately  across  the  DoD.  The  problems  with  the  government’s 
current  data  repositories  have  plagued  the  DoD  for  years.  Poor  data  quality,  incompatible 
data  formats,  inconsistent  reporting  timeliness,  lack  of  data  entry,  diffused  accountability, 
and  subjective  evaluations  are  among  the  issues  associated  with  the  way  the  DoD  has 
collected,  analyzed,  and  reported  past  performance  information. 


2  However,  it  should  be  noted  that  while  all  federal  acquisitions  are  required  to  use  the  PPIRS,  not  all 
government  organizations  have  complied. 
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Proposed  Solutions 

Implement  and  Enforce  DoD  Policies  for  Past  Performance 

The  OFPP  and  the  Defense  Procurement  and  Acquisition  Policy  (DPAP)  Directorate 
in  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(OUSD[AT&L])  have  recently  issued  guidance  and  policy  that  emphasize  the  importance  of 
information  on  past  performance.  Current  policies,  however,  lack  appropriate  enforcement 
mechanisms  to  make  this  policy  effective  and  actionable.  The  OUSD[AT&L]/DPAP  should, 
therefore,  issue  a  DoD-wide  mandate  to  populate  past  performance  databases  in  a  timely 
and  consistent  manner.  Per  FAR  (2010)  Part  42.15,  the  responsibility  for  reporting  past 
performance  resides  with  the  contracting  officer,  but  it  is  important  to  make  this  a  shared 
responsibility  between  the  PMO  and  the  contracting  office.  The  program  team  (e.g.,  program 
manager,  contracting  officer’s  technical  representative  [COTR])  should  evaluate  how  well 
the  contractor  is  performing  against  the  SOW  requirements.  The  contracting  office  should 
collect  this  information  from  the  PMO  and  also  provide  input  on  the  contractor’s  performance 
against  the  contract  requirements  (e.g.,  reporting  requirements,  terms  and  conditions).  The 
contracting  office  should  have  ultimate  responsibility  for  collecting  this  information  and 
populating  the  past  performance  database  in  a  timely  and  consistent  manner. 

The  DoD  should  increase  enforcement  of  this  policy  through  a  variety  of 
mechanisms.  Agencies  should  be  required  to  report  metrics  on  how  well  they  are  complying 
with  this  mandate.  The  OUSD[AT&L]/DPAP  should  collect  these  metrics  on  a  monthly  basis 
and  publish  the  results  on  the  DPAP’s  website.  The  head  of  each  Combatant  Command, 
Service,  or  Agency  (C/S/A)  should  be  held  accountable  for  C/S/A-wide  compliance  with  this 
policy. 

Additionally,  the  DoD  should  require  that  past  performance  reviews  become  a 
mandatory  part  of  Program  Executive  Office  (PEO)  acquisition  program  reviews.  This  will 
ensure  that  program  offices  are  held  accountable  for  meeting  this  policy  requirement.  As  a 
further  incentive,  the  DoD  should  prohibit  programs  from  exercising  option  awards,  award 
terms,  or  award  fee  payouts  until  they  have  populated  past  performance  into  a  database. 

Lastly,  the  DoD  should  include  an  evaluation  of  compliance  with  this  process  as  part 
of  each  contracting  officer’s  annual  pay  and  performance  review.  Metrics  that  report 
compliance  with  this  mandate  at  the  individual  contracting  officer  level  will  further  enhance 
the  enforcement  of  this  policy. 

Identify  Quantifiable  Metrics  for  Agency-Wide  Reporting 

Past  performance  evaluations  today  are  based  entirely  on  subjective  judgments  by 
the  evaluator.  This  makes  it  difficult  to  compare  performance  across  programs  and 
contractors.  Additionally,  observations  show  that  program  managers  and  contracting  officers 
are  often  reluctant  to  report  negatively  on  past  performance  because  this  can  reflect  poorly 
on  their  own  ability  to  manage  the  program  or  contract.  Furthermore,  to  avoid  conflict  with 
the  contractor,  the  government  may  refrain  from  documenting  performance  deficiencies  in 
official  databases.  As  a  result,  the  past  performance  write-up  does  not  always  reflect  a 
contractor’s  performance  accurately. 

The  DoD  should  agree  on  a  set  of  quantifiable  metrics  to  capture  in  data  fields  within 
databases  of  past  performance.  The  use  of  such  quantifiable  and  simple  yes/no  metrics  can 
provide  a  consistent  and  reliable  way  to  compare  contractor  performance  across  contracts. 
For  example,  earned  value  management  (EVM)  data  can  be  used  to  report  cost  and 
schedule  performance  on  complex  acquisitions.  Award  fee  determinations  can  also  provide 
a  metric,  since  this  is  a  bilateral  agreement  and  contractors  often  use  this  metric  to  measure 
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their  own  performance.  Simple  yes/no  metrics  can  be  used  for  non-EVM  contracts.  Such 
questions  could  include  the  following:  Did  the  contractor  perform  within  proposed  cost?  Did 
the  contractor  deliver  on  time?  Did  the  program  encounter  a  Nunn-McCurdy  breach?  Each 
of  the  metrics  and  questions  can  be  followed  by  a  data  field  that  allows  the  government  to 
explain  the  evaluations  provided. 

Include  Validated  and  Approved  Contract  Deliverables  Into  Past  Performance 

Databases 

The  DoD  can  increase  the  quantity  and  quality  of  the  data  in  past  performance 
databases  by  incorporating  information  on  validated  and  approved  contract  deliverables.  For 
example,  summary  narratives  on  award  fees  provide  a  wealth  of  information  on  how  the 
contractor  performed  during  the  award  fee  period.  As  noted  above,  award  fees  reflect 
consensus  between  the  government  and  contractor  on  the  factors  that  contributed  to  the 
final  fee.  This  type  of  information  can  be  very  useful  during  evaluations  of  past  performance, 
since  it  gives  the  DoD  a  more  complete  picture  of  how  well  the  contractor  performed  over 
the  life  of  a  contract. 

Lastly,  past  performance  databases  should  include  a  mechanism  for  uploading 
documents  directly  into  CPARS  to  share  with  other  government  counterparts.  This  would 
greatly  enrich  the  quality  and  quantity  of  past  performance  data. 

Conclusion  and  Recommendations 

The  lack  of  adequate  past  performance  data,  tools,  and  processes  hinders  the 
government  from  effectively  evaluating  the  qualifications  of  companies  and  key  personnel. 
The  DoD  tends  to  deemphasize  past  performance  evaluations  during  the  competitive  source 
selection  process,  largely  because  data  that  would  allow  it  to  discriminate  among  proposals 
is  unavailable.  Current  government  tools  have  proven  inadequate  to  collect,  analyze,  and 
report  information  on  past  performance.  The  absence  of  timely,  reliable,  documented  data 
impedes  the  entire  cycle  of  collecting,  reporting,  and  evaluating  past  performance  during 
source  selection  (see  Figure  1).  The  inability  to  use  past  performance  as  a  predictor  of 
future  contract  success  places  the  DoD  at  risk  of  repeating  program  mistakes. 


Past  Performance  Cycle 


Figure  1.  Past  Performance  Cycle 


Evaluating  a  company’s  performance  record  allows  the  government  to  assess  the 
effectiveness  of  a  contractor’s  management  processes,  tools,  and  resources.  Because 
effective  performance  depends  on  qualified  and  experienced  personnel  who  know  how  to 
replicate  success,  the  ability  to  evaluate  proposed  key  personnel  also  becomes  an  important 
aspect  of  estimating  a  company’s  ability  to  meet  the  contract  requirements.  Yet  the  DoD 
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also  lacks  the  necessary  processes  to  assess  the  capabilities  of  the  key  personnel  proposed 
on  a  contract. 


The  DoD  currently  lacks  a  comprehensive  approach  to  tackle  the  issues  identified  in 
this  paper.  Since  the  problems  are  interdependent,  they  require  a  holistic  approach  that 
addresses  the  tools,  processes,  and  policies  surrounding  the  past  performance  cycle 
pictured  in  Figure  1 .  The  processes  will  not  improve  without  adequate  tools  to  provide  the 
data.  They  will  not  be  effective  without  the  proper  policies  in  place  to  enforce  them,  and  they 
cannot  be  implemented  without  the  proper  processes  and  tools  to  support  them. 

The  recommendations  summarized  in  the  following  list  have  been  brought  forward 
throughout  this  paper  as  important  steps  toward  improving  the  DoD’s  probability  of 
achieving  successful  future  program  outcomes: 

1 .  Use  alpha  contracting  negotiations  with  pre-qualified  vendor  candidates. 
Amend  FAR  Part  15  to  allow  for  a  QBS-like  approach  across  a  broader  range 
of  acquisitions. 

2.  Use  product  demonstrations  as  part  of  source  selection  to  increase  accuracy 
in  evaluations  of  the  contractor’s  performance  record. 

3.  Include  oral  presentations  in  the  evaluation  process;  improve  the  consistency 
and  depth  of  personnel  evaluations  by  using  the  Q&A  process  to  verify  the 
qualifications  and  experience  cited  in  proposed  key  personnel  resumes. 

4.  Issue  a  policy-wide  mandate  that  enforces  past  performance  data-entry 
standards.  Use  an  online  tool  to  track  and  report  compliance  to  this  policy 
requirement. 

5.  Agree  on  a  set  of  quantifiable  metrics  to  capture  in  data  fields  within 
databases  of  past  performance. 

6.  Increase  the  quantity  and  quality  of  data  in  past  performance  databases  by 
incorporating  information  on  validated  and  approved  contract  deliverables. 
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BLUF 

Selecting  the  right  contractor  is  perhaps  the  most  important 

contributor  to  program  success. 

The  Past  Performance  as  an  Indicator  of  Future 
Performance  initiative  is  a  MITRE  Project  to  improve  the 
probability  of  a  successful  program  outcome. 

Emphasize  evaluating  the  contractor’s  record  of  performance  vs. 
evaluating  the  contractor’s  proposed  solution. 

-  Initiate  consistent  and  thorough  processes  to  effectively  evaluate 
proposed  key  contractor  personnel. 

Improve  the  accuracy  and  availability  of  government  of  past 
performance  data  repositories. 

The  recommendations  can  benefit  the  PMO  by  providing 
repeatable  data  driven  processes  using  historically  disparate 
best  practices  that  can  be  effectively  applied  to  past 
performance. 
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The  government  must  strive  to  enter  into  business  relationships  that  provide 

high  confidence  of  success. 
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Past  Performance  Project  Objectives 

Provide  approaches  to  improve  the  probability  of  selecting  an  industry 
partner  to  maximize  the  probability  of  program  success . 

■  Appraise  the  contractor’s  record  of  performance. 

Increase  Alpha  contracting  negotiations  with  pre-qualified  candidates. 

Consider  product  demonstrations  that  exhibit  performance  standards  early  in 
the  source  selection  process. 

■  Evaluate  proposed  key  contractor  personnel. 

Require  oral  presentations  using  an  extensive  question  and  answer  process  to 
verify  “actual”  qualifications  and  experience. 

Implement  behavioral  interviewing  techniques  to  identify  the  desired  attributes 
of  key  personnel’s  past  performance. 

■  Improve  timeliness  and  quality  of  past  performance  data  repositories. 

-  Enforce  past  performance  data  entry  standards. 

-  Improve  data  repository  tools  to  include  automated  quantifiable  metrics. 


Evaluating  contractor  past  performance  is  an  important  step  to  creating  a 
genuine  business  partnership  between  the  government  and  a  contractor. 
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Background 

■  Federal  Acquisition  Streamlining  Act  of  1994. 

Congress  acknowledged  that  it  is  appropriate  and  relevant  for  the  government 
to  consider  a  contractor’s  past  performance  in  evaluating  whether  that 
contractor  should  receive  future  work. 

The  Office  of  Federal  Procurement  Policy  (OFPP),  May  2010. 

Directed  government  agencies  to  integrate  past  performance  data  systems, 
Contractor  Performance  Assessment  Reporting  System  (CPARS)  and  the  Past 
Performance  Information  Retrieval  System  (PPIRS). 

■  Administrator  for  Federal  Procurement  Policy  Memorandum, 

Daniel  I.  Gordon,  January  21,  2011. 

“...ensuring  that  Contracting  Officers  have  access  to  meaningful  past 
performance  assessments  is  so  important  to  improving  source  selection 
decisions  that  we  want  to  do  everything  we  can  to  improve  both  the  quantity 
and  quality  of  past  performance  assessments  in  PPIRS.” 


The  policies  and  mandates  lack  enforcement  mechanisms  to  incentivize  better 

behavior  and  improve  past  performance  practices. 


5 

©  2012  The  MITRE  Corporation.  All  rights  reserved 


MITRE 


Process  Overview 


Past  Performance  Cycle 
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Contractor's  Record  of  Performance 


Past  Performance  Cycle 
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Contractor  s  Record  of  Performance 

(Weakness) 

The  DoD  currently  lacks  both  adequate  data  and  effective  processes  to 
evaluate  a  company's  past  performance  qualifications . 

■  Government  Accountability  Office  Study,  April  2009. 

Analyzed  62  procurements  across  5  agencies  (DoD,  DOE,  GSA,  DHS,  and 
NASA). 

82%  of  past  performance  evaluations  did  not  contain  narratives  sufficiently 
detailed  to  establish  credible  or  justifiable  ratings. 

60%  of  the  Contracting  Officers  stated  that  past  performance  is  rarely  or  never 
the  deciding  factor  in  selecting  a  contractor. 

■  However,  the  study  concluded... 

Contracting  officials  agreed  that  for  past  performance  to  be  meaningful  in 
contract  award  decisions  it  must  be  documented,  relevant,  and  reliable. 

Acquisitions  that  placed  emphasis  on  past  performance  noted  that  this 
encouraged  companies  to  perform  better. 

Without  accurate  past  performance  information  the  government  must  rely 
heavily  on  technical  and  cost  factors  that  describe  a  hypothetical  situation. 


Raise  the  value  of  “past  performance”  in  the  source  selection  process 
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Contractor  s  Record  of  Performance 

(Solution) 

■  Alpha  contracting  negotiations  with  pre-qualified  vendors. 

-  Utilizing  the  best  from  the  Qualifications  Based  Selection  (QBS)  (FAR  Part  36) 
to  improve  innovation,  quality,  and  creativity: 

The  government  determines  the  desired  qualifications,  experience  and 
demonstrated  competence  of  interested  vendors. 

Then  creates  a  short  list,  ranking  the  bidders  by  their  qualifications. 

The  government  conducts  interviews  with  the  firms  on  the  short  list  and  then 
re-ranks  the  firms. 

4  Finally,  the  government  negotiates  a  statement  of  work  (SOW)  and  fair  price  with 
the  most  qualified  vendor  based  on  a  Government  Independent  Cost  Estimate. 

Much  of  QBS  can  be  applied  to  the  advisory  multi-step  process  (FAR  Part 
15.202)  based  entirely  on  past  performance  criteria  to  narrow  down  the  pool. 

Use  a  Q&A  session  as  part  of  an  oral  presentation  strategy. 

Performance  improvements  include  enhancing  communication,  refining  and 
clarifying  requirements,  and  bounding  the  technical  solution  by  the  capabilities 
of  the  contractor. 


Amend  FAR  Part  15  to  allow  DoD  to  directly  benefit  from  the  QBS  process 
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Contractor  s  Record  of  Performance 

(Solution) 


Pre-qualification  product  demonstrations. 

Supplements  limited  availability  of  past  performance  information  and  verifies 
the  relevance  of  previous  experience. 

Demonstrations  can  include  any  material  representation  of  similar  conceptual 
past  experience,  such  as  prototypes  or  software  demonstrations. 

This  source  selection  strategy  assists  with  initial  narrowing  the  pool  of  qualified 
vendors. 

Establishes  recent  and  relevant  experience. 


Demonstrations  become  tangible  evidence  of  past  performance. 
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Key  Contractor  Personnel 


Past  Performance  Cycle 
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Key  Contractor  Personnel 

(Weakness) 

DoD  lacks  consistent  and  thorough  processes  to 
evaluate  key  contractor  personnel. 

■  Limited  evaluation  of  proposed  key  contractor  personnel. 

DoD  processes  should  aspire  not  only  to  select  the  strongest  contractor,  but 
also  to  ensure  that  the  contractor  assigns  the  “A  Team”  to  the  program. 

Most  source  selection  teams  limit  evaluation  of  key  personnel  to  a  simple 
checklist  to  whether  they  meet  minimum  experience  criteria. 

Current  minimal  use  of  oral  presentations  limit  face-to-face  access  to  key 
personnel  and  an  opportunity  to  assess  group  dynamics. 


When  companies  know  their  key  personnel  will  be  evaluated,  they  have 

an  incentive  to  offer  their  best  performers. 
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Key  Contractor  Personnel 

(Solution) 


Qualifications  of  the  personnel  proposed  for  key  roles  on  the  contract 
are  paramount  to  the  future  success  of  the  program. 

Evaluate  proposed  key  contractor  personnel. 

Require  oral  presentations  using  an  extensive  question  and  answer  (Q&A) 
process  to  verify  “actual”  qualifications  and  experience. 

Implement  behavioral  interviewing  techniques  to  identify  the  desired  outcomes 
of  key  personnel’s  through  actual  past  performance. 

70%  of  Fortune  500  companies  credit  behavioral  interviewing  with  being  55% 
predictive  of  future  on-the-job  behavior. 


Behavioral  interviewing  techniques  illuminate  the  actual 
past  performance  accomplishments  of  key  personnel. 
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Data  Repositories 


Past  Performance  Cycle 
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Data  Repositories 

(Weakness) 


DoD  Lacks  the  adequate  tools  and  information  to  collect,  analyze,  and 
report  past  performance  information  consistently  across  contracts. 

■  Despite  recent  improvements  to  systems  and  policies  current  data 
repositories  continue  to  perform  their  functions  inadequately. 

-  Lack  of  enforcement  of  past  performance  policies. 

-  Limited  quantifiable  metrics  for  agency-wide  reporting. 

-  Lack  of  validated  and  approved  contract  deliverables  in  the  past 
performance  databases. 


A  high-quality  data  repository  greatly  strengthens  the  government's  ability  to 
evaluate  past  performance  as  part  of  the  source  selection  process. 


15 

©  2012  The  MITRE  Corporation.  All  rights  reserved 


MITRE 


Summary  of  Agencies'  Past  Performance  Assessment  Review 


Schedule  Control 


Quality  of  Product/Service 


NASA  HHS  DoD  DoT  State  DoE  DHS  VA  GSA  DoJ 
Agency 


NASA  DoT  HHS  DoD  DoE  State  DHS  VA  DoJ  GSA 
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HHS  DcT 


DoD  State  DoE  VA 
Agency 


DHS  GSA  DoJ  NASA 


□  58.4 

□  53.3 

□  50.0 

□  48.8 

■  43.2 

□  39.5 

■  33.0 

□  27.0 

■  15.4 

■  4.3 


Source:  Gordon,  Daniel,  Improving  Contractor  Past  Performance  Assessments  Memo  ,  OFPP,  January  21,  2011 


Average  High  59%  and  Low  13% 
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PAST  PERFORMANCE 

INFORMATION  RETRIEVAL  SYSTEM 


DoD  Reporting  Compliance  Metrics  2Q  FY2012 


PERCENTAGE 

PERCENTAGE 

AGENCY  NAME 

CONTRACTS 

CONTRACTS 

QTR  TO  QTR 

COMPLETED 

COMPLETED 

DELTA 

3  JAN  2012 

6  APR  2012 

DEPT  OF  THE  NAVY 

68.8 

46.8 

22- 

DEPT  OF  THE  ARMY 

53.7 

48.3 

5.4- 

DEPT  OF  THE  AIR  FORCE 

84.6 

57.3 

27.3- 

DEFENSE  HUMAN  RESOURCES  ACTIVITY 

87.5 

83.3 

4.2- 

TRICARE  MANAGEMENT  ACTIVITY  (TMA) 

72.7 

29.2 

43.5- 

DEFENSE  THREAT  REDUCTION  AGENCY  (DTRA) 

77.3 

52.5 

24.8- 

DEFENSE  CONTRACT  MANAGEMENT  AGENCY  (DCMA) 

69.1 

22.4 

46.7- 

USTRANSCOM 

83.9 

11.7 

72.2- 

BUSINESS  TRANSFORMATION  AGENCY 

31.5 

4.5 

27- 

DEFENSE  ADVANCED  RESEARCH  PROJECTS  AGENCY  (DARPA) 

73.6 

38.8 

34.8- 

DEFENSE  INFORMATION  SYSTEMS  AGENCY  (DISA) 

70.5 

31.6 

38.9- 

DEFENSE  LOGISTICS  AGENCY 

29.6 

24.6 

5- 

DEFENSE  SECURITY  COOPERATION  AGENCY 

7.2 

4.4 

2.8- 

DEFENSE  SECURITY  SERVICE 

42.8 

6.4 

36.4- 

DEFENSE  COMMISSARY  AGENCY  (DECA) 

5.6 

5.1 

.5- 

DEFENSE  FINANCE  AND  ACCOUNTING  SERVICES  (DFAS) 

84.1 

42.3 

41.8- 

DEFENSE  MEDIA  ACTIVITY  (DMA) 

13.1 

2.7 

10.4- 

DEPT  OF  DEFENSE  EDUCATION  ACTIVITY  (DODEA) 

4.1 

0 

4.1- 

WASHINGTON  HEADQUARTERS  SERVICES  (WHS) 

21.4 

5.1 

16.3- 

UNIFORMED  SERVICES  UNIVERSITY  OF  THE  HEALTH  SCIENCES  (USUHA) 

9.1 

0 

9.1- 

MISSILE  DEFENSE  AGENCY  (MDA) 

33.3 

13.7 

19.6- 

U.S.  SPECIAL  OPERATIONS  COMMAND  (USSOCOM) 

59.9 

16.5 

43.4- 

TOTAL  AVERAGES 

59% 

47% 

12%- 

Source:  Ginman,  Richard,  Contractor  Past  Performance  Assessment  Reporting-2nd  Quarter  2012  Memo,  April  26,  2012 
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Data  Repositories 

(Solution) 

Implement  and  enforce  DoD  policies  for  past  performance . 

Make  the  responsibility  for  reporting  past  performance  a  shared  responsibility 
between  the  contracting  office  and  the  program  management  office. 

The  contracting  office  should  have  ultimate  responsibility  for  collecting  and 
populating  the  past  performance  database  in  a  timely  and  consistent  manner. 

Agencies  should  be  required  to  report  compliance  metrics  against  this  mandate. 

DoD  should  further  require  that  past  performance  reviews  become  a  mandatory 
part  of  Program  Executive  Office  (PEO)  acquisition  program  reviews. 

DoD  should  prohibit  programs  from  exercising  option  awards,  award  terms,  or 
award  fee  payments  until  they  have  populated  past  performance  data  into  the 
database. 

Lastly,  DoD  should  include  an  evaluation  of  compliance  with  this  process  as  part  of 
each  contracting  officer’s  annual  pay  and  performance  review. 


Follow-up,  measure,  and  enforce  timely  and  accurate  past  performance  data  management. 
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Data  Repositories 


(Solution) 


Identify  quantifiable  metrics  for  agency-wide  reporting. 

Increase  the  accuracy  of  past  performance  evaluations. 

Develop  objective  measures  that  can  be  fairly  applied  to  all  contractors 
allowing  consistent  performance  comparisons  across  programs  and 
contractors. 

Provide  safeguards  for  program  managers  and  contracting  officers  from 
contractor  retaliation  through  multiple  source  evaluation  data  gathering  that 
encourages  honest  and  accurate  past  performance  evaluation  reporting. 

DoD  would  benefit  from  the  use  of  quantifiable,  and  simple  yes/no 
metrics  to  avoid  subjective  judgments,  and  introduce  consistency 
across  agencies. 


MITRE 
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Data  Repositories 

(Solution) 


Provide  access  to  validated  and  approved  contract  deliverables 

in  the  past  performance  database. 


The  DoD  can  increase  the  quantity  and  quality  of  the  data  in  past  performance 
databases  by  incorporating  information  on  validated  and  approved  contract 
deliverables. 

Provides  a  more  complete  picture  of  how  well  the  contractor  performed  over 
the  life  of  a  contract. 


Past  performance  databases  should  include  a  mechanism  for  uploading 
documents  directly  into  CPARS  to  share  with  other  government  agencies. 
This  would  greatly  enrich  the  quality  and  quantity  of  past  performance  data. 


Raise  the  value  of  “past  performance”  in  the  source  selection  process 
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Summary 

The  lack  of  adequate  past  performance  data,  tools,  processes,  and  policy 
enforcement  hinders  the  government  from  effectively  evaluating  the  qualifications 

of  companies  and  key  personnel . 


Past  Performance  Cycle 
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Recommendations 

Implementation  of  the  following  recommendations  could  have 
a  significant,  positive  impact  on  future  program  success. 

Use  alpha  contracting  negotiations  with  pre-qualified  vendor  candidates.  Amend  FAR 
Part  15  to  allow  for  a  QBS-like  approach  across  a  broader  range  of  acquisitions. 

Use  product  demonstrations  as  part  of  source  selection  to  increase  accuracy  in 
evaluations  of  the  contractor’s  performance  record. 

Include  oral  presentations  in  the  evaluation  process;  improve  the  consistency  and  depth 
of  personnel  evaluations  by  using  the  Q&A  process  to  verify  the  qualifications  and 
experience  cited  in  proposed  key  personnel  resumes. 

Issue  a  policy-wide  mandate  that  enforces  past  performance  data  entry  standards.  Use 
an  online  tool  to  track  and  report  compliance  to  this  policy  requirement. 

Agree  on  a  set  of  quantifiable  metrics  to  capture  in  data  fields  within  databases  of  past 
performance. 

Increase  the  quantity  and  quality  of  data  in  past  performance  databases  by 
incorporating  information  on  validated  and  approved  contract  deliverables. 


Past  performance's  value  will  increase  with  efficient  tools  to  provide  the  data, 
effective  enforcement  of  policies,  and  supportinq  processes. 
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Past  performance  is  an  indicator  of  future  performance 


Provides  approaches 
to  improve  the  probability 
of  selecting  an  industry 
partner  to  maximize  the 
probability  of  program 
success 
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